Time-dependent Nonequilibrium Thermodynamics: 
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Master equation could be applied to model various kinds of biochemical systems. A general 
theory for its time-dependent nonequilibrium thermodynamics is rigorously derived. We not only 
introduce a concept of general internal energy, but also propose an extension of the equilibrium 
state. Moreover, we find out an explicit expression for the extended form of the Second Law, 
which is phenomenologically stated as "the conversion of work to excess heat is irreversible". The 
theory is carefully applied to several thermodynamic processes, and its consistence with the classical 
thermodynamics is clearly shown. 
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I. INTRODUCTION 

Equilibrium thermodynamics emerges when Carnot 
proposed the first theoretical treatise on mechanical work 
and efficiency in heat engines in the early nineteenth cen- 
tury. However, one hundred fifty years after its formu- 
lation, the second law of thermodynamics still appears 
more as a program than a well defined theory, and all 
the thermodynamic potentials could be well-defined only 
in equilibrium states. This is one of the main reasons why 
the equilibrium thermodynamics could hardly be applied 
to real biochemical systems, because living cells must 
continually extract energy from their surroundings in or- 
der to sustain the characteristic features of life such as 
growth, cell division, intercellular communication, move- 
ment and responsiveness to their environment. 

The researches on irreversible systems far from equi- 
librium began with the works by Haken [l|, 0] about laser 
and Prigogine, etc. 0] about oscillations of chemical 
reactions. Prigogine, I. and his collaborators provided 
explicit expressions for entropy production in various sit- 
uations, and regarded a nonequilibrium steady state as a 
stationary open system with positive entropy production 
rate 

In 1998, Oono and Paniconi |5| proposed a framework 
of steady state thermodynamics, and distinguished the 
steadily generated heat which is generated even when 
the system remains in a single steady state and the total 
heat. They called the former the "housekeeping heat", 
which is equal to the entropy production in steady state 
and may come from the chemical driven force in biochem- 
ical systems [1, 0] ■ The key point of their work is that "if 
we can carefully remove the steadily produced heat due 
to housekeeping dissipation, then the state should not be 
very different from equilibrium" . Moreover, they also put 
forward a phenomenological extended form of the Second 
Law: "A process converting work into excess heat is ir- 
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reversible. And 'reversibility' is modulo house-keeping 
heat, which is produced anyway". 

On the other hand, it has been known for several 
decades that one can use stochastic processes as mathe- 
matical tools to study nonequilibrium states and steady 
cycle fluxes. In 1953, Onsager and Machlup [1, |^ pro- 
posed the Onsager-Machlup principle, which is actu- 
ally a functional formula about the probability density 
of a stochastic process close to equilibrium. T.L. Hill 
[13, [m, [H, [il] and J. Schnakenberg [l4j successfully con- 
structed a general mesoscopic master-equation model for 
biochemical systems and investigated its thermodynamic 
properties far from equilibrium. Since then, a rather 
complete mathematical theory for nonequilibrium steady 
states has been developed for stochastic models [l5| . 

In recent years, a few interesting relations that de- 
scribe the statistical dynamics of driven systems even 
far from equilibrium have been discovered, including 
the fluctuation theorems of sample entrop y p roduction 
EEHIliLJarzynski's equality HlU,!!!, Hatano- 
Sasa equality 23|, etc. The main purpose of Hatano and 
Sasa's work [23| is to derived the first explicit expression 
for the extended form of the Second Law of Thermo- 
dynamics, namely TAS > Qex, where S is the general 
entropy defined in their paper, and Q^x is the excess heat. 

In the present paper, we start with the definition of 
Gibbs entropy [2J| for stochastic processes, and accept 
the theory of entropy production along a single stochas- 
tic trajectory [2^. Then, following the definitions of the 
three kinds of heat [1^ [2y] , it is found out the housekeep- 
ing heat is always dissipated towards the environment, 
unless the corresponding steady state is at equilibrium. 
Furthermore, the dissipative work subtracting the excess 
heat along the trajectory does not depend on the particu- 
lar "path" taking through the parameter space, namely, 
only depends upon the initial and final states. Thus, 
there exactly exists a "general internal energy", whose 
derivative is just the difference of the dissipative work 
and excess heat. 

Regarding the Second Law of Thermodynamics, it 
is shown that the formulation here not only accords 
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well to its former form, namely the nonnegativity of 
entropy production, but also put forward another ex- 
plicit mathematical interpretation for its extended form, 
which is phenomenologically stated as "the conversion 
of work to excess heat is irreversible" d, HI]- The ex- 
tended form says that the entropy production rate af- 
ter subtracting the house-keeping heat is still nonnega- 
tive, namely T ■ epr{t) — Qhk{t) > 0, which consequently 
gives rise to the extended forms of the Clausius inequality 
and Helmholtz/Gibbs free energy inequalities, and takes 
Hatano-Sasa's work as a special example. 

Similar result has been put forward in Langcvin sys- 
tems p?! , but the analysis here is much more compre- 
hensive. 



II. THEORY 

Now stochastic models are widely used in physics, 
chemistry, biology and even in economics. The master- 
equation process discussed in this article could be applied 
to model chemical reactions, which are of special inter- 
est in biology, in relation with their coupling with ac- 
tive transport across membrane [Tp. |28l| and also recent 
mechanisms of molecular motors [29| . Furthermore, in 
real biochemical systems, the external parameters such 
as the concentrations of external signal proteins always 
oscillate or remarkably fluctuates, which stimulates the 
necessity for the analysis of time-dependent processes. 

In physics, master equation is a set of first-order dif- 
ferential equations of the mesoscopic system: 



A. Essential Notations and Fundamental Relations 

Equilibrium thermodynamics is a generalization of me- 
chanics by introducing three new concepts: equilibrium 
states, internal energy and entropy which related to dis- 
order. 

The concept of states is an ensemble of well-defined 
variables for identifying any property of the system un- 
der study. Nowadays, whether the most important quan- 
tities in thermodynamics, such as temperature, volume, 
pressure, etc. could be generalized to arbitrary nonequi- 
librium states or not is still unclear. Fortunately, the 
situation is quite different when we study the master- 
equation processes, in which the state variables are just 
their transient distributions and speciflc time-dependent 
transition laws. 

However, the notation of internal energy seems un- 
able to be directly generalized to arbitrary state, and the 
First Law of Thermodynamics remains unknown. In the 
present article, we will show that there does exist an ob- 
vious way to generalize it to the time-dependent Markov 
processes, enlightened by the mathematical equivalence 
of Jarzynski and Hatano-Sasa equalities [2^. |33|. 

In the present article, we only investigate the isother- 
mal system with fixed temperature T. The role of tem- 
perature here is simply to keep the environment constant, 
therefore we did not have to be so careful about precise 
parameterization of the temperature. We simply used the 
same temperature scale as the heat baths, and declared 
that the temperature may be measured by a thermome- 
ter, just as Sasa claimed in his recent review [3^ . 



1. Gibbs entropy postulate and its generation 



describing the dynamical evolution of a probability dis- 
tribution Pi(t) over states i = 1, 2, • • ■ , TV. The quantity 
qij{t) is the transition density (probability per time) to 
state j from state i. It contains internal rate constants 
as well as external conditions imposed by the coupling to 
the reservoir systems. The basic properties of this model 
has been discussed previously [30| • 

One could take a stationary Markov chain, in which 
the probability transition density Q{t) is invariant, as 
the mathematical model of the combination and trans- 
formation of biochemical polymers [111, [13] ■ Each state 
of the Markov chain corresponds to a mesoscopic state of 
polymers. 

Let us also mention that the number N need not to 
be finite, and the system could also be regarded as the 
stochastic model of coupled chemical reactions (chemical 
master equation) [ij, HH ■ 



Unlike the traditional approach of equilibrium thermo- 
dynamics, we should start with the general definition of 
entropy, and then turn back to consider the existence of 
internal energy. The common definition of Gibbs entropy 
associated with any discrete probability distribution {pi} 
is 

i 

where k is the Boltzmann constant. 

In statistical mechanics, it gives the entropy for a 
canonical ensemble of a molecular system at constant 
temperature, and is a generalization of Boltzmann's for- 
mula to a situation with nonuniform probability distri- 
bution. 

Denote the real distribution at time t is p{t) = {pi(t)}, 
and we define the general entropy at time t as S{t) = 

It is widely known that the entropy c hang e dS could 
be to distinguished in two terms [j, 0, [ssf : the first, 
deS is the transfer of entropy across the boundaries of 
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the system, and the second diS is the entropy produced 
within the system. 

Here, it is easy to derive that 0, Issj ] 



dS{t) 
dt 



= d,S + deS ^ epr{t) - hdr{t), (2) 



where 



epr{t) = = ^k^{pi{t)q^j{t)-pj{t)qj,{t)) log 



Pj{t)qji{t) 



ah the trajectories, then one gets epr{t) = {epr{X{t),t)) 
and hdr{t) = {hdr{X{t),t)). 

It is also pointed out by Seifert [1^ that the trajectory- 
dependent entropy of the particle could be measured 
experimentally for a time-dependent protocol by first 
recording over many trajectories the probability distri- 
bution Pi{t)] from which the entropy of each trajectory 
can be inferred. 



is just the instantaneous entropy production rate [30|, 
and 



hdr{t) = deS = ^k^{p,{t)qij{t) - pj{t)q^i{t))\og 



qji{t) 



is due to the exchange of heat with the exterior, called 
the heat dissipation rate. 



2. Entropy production along a stochastic trajectory 

A trajectory point of view will help us to realize the 
stochastic nature of kinetics profoundly. Regarding its 
stochastic behavior, any individual particle of the en- 
semble is governed entirely by the transition density Q(t) 
of the system, and ensemble properties depend on their 
statistics. 

We write the trajectory (sample path) of the Markov 
chain as X{t). Denote by nx the number of times that 
X jumps in the time interval [0,T]. Let Tq = 0, Ti — 
M{t > : Xit) ^ XiO)}, Tk = mi{t > Tu-i : X{t) ^ 
X(Tfc_i)|, a,nd Tm^+i = T be the jumping times. 

Seifert [25[ defined the sample trajectory entropy along 
the trajectory at time t as S{X{t),t) — — k log p{X{t),t), 
recalling p{t) is the real distribution of the Markov chain 
{X{t)} at time t. 

The equation for the motion of S{X{t),t) becomes 



3. Decomposition of heat conduction 

The idea of decomposing the total heat into a "house- 
keeping" part and another "excess" part was put forward 
by Oono and Paniconi Q , and made explicit in Langevin 
systems by Hatano and Sasa [23j . 

The sample heat dissipation hdr{X{t),t) could be re- 
garded as the total heat conduction Qtot{X{t),t) with 
the medium, i.e. 

QtotiXit),t)^T-hdriXit),t), 

and its ensemble average Qtot{t) = T ■ hdrit). By con- 
vention, we take the sign of heat to be positive when it 
flows from the system to the heat bath. 

For any fixed i, there is a steady distribution 7r(t) ~ 
{7ri(i)} corresponding to Q{t) satisfying Ti{t)Q{t) = 0, 
which need not obey the detailed balance condition 
■ni{t)qij{t) = ■nj{t)qji(t). 

Then we could define the other two kinds of heat: the 
housekeeping heat and excess heat along the trajectory 



Qhk{X{t),t) = -kTj2St=Tjog 



'^X(Tk){t)qx(Tk)X{Tk-i){t) 



dS{X{t),t) 
Jt 



where 

epr{X{t),t) = -k 



= epr{X{t), t) - hdr{X{t), t), 

d\ogp{X(t),t) 
dt 



Qe. {X{t) , t) = fcT V 5t=T, log ^^(I±L , 
and obviously Qtot{X{t),t) = Qhk{X{t),t) 

Qex{X{t),t). 

After averaging over all the trajectories, one gets 



1 Px{n){t)qxin)xin_,){t) 

-k > dt^Tk log 7- 7-. 

Px(n_,){t)qx(T,_,)xiT,)(t) 



^,3 



is the sample entropy production at time t, and 

qx(n)x{n-i){t) 



hdr{X{t), t) = -kY^ 6t=Tk log 



Qhk{t) = \kTy^{p,{t)q,,{t)-pj{t)qj^{t))\og- 



Tri it)qijit) 

{i)qJ^{ty 



qx{n-i)x{n){t) 



is the sample heat dissipation in the medium. 

The rationality after these identifications for the 
change rate of entropy becomes clear when averaging over 



and also Qtotit) = Qex{t) + Qhk{t)- 

More importantly, we found out that the housekeeping 
heat is always nonnegative, which implies the nonequilib- 
rium essence of the system: 
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states, i.e. 



Qhkit) = kTJ2p^it)q^3it)\0g 



T^i{t)qij{t) 

nj{t)qj,{t) 



> 



-1) 



"TTi {t)qij{t) 

0, 



T4^(X(t), i) - Qe.{X{t), t) = dU{X{t),t), 

where U{X{t),t) ~ ~kT log Trx(t){t) could be regarded 
as the internal energy of state X{t). 

Thus, there exactly exists a "general internal energy" 
U{t) = {U{X{t),t)) -fc^E,P^(^)log^^(^), whose 
derivative is just the difference of the dissipative work 
and excess heat, i.e. 



by making use of a simple inequality logx < x — 1 for 
a; > 0, and the identity J^j Qiji^) = 0- 

For equilibrium system, Q^x reduces to the total heat 
Qtot, because in this case Qhk = due to TTi{t)qij{t) = 
nj{t)qji{t). And in time-independent steady state, 
Qex{i) = 0, and hence the housekeeping heat Qhk equals 
the work done by the external driven force, which is all 
dissipated [tI. Issj. 

However, the situation is quite different for the time- 
dependent nonequilibrium system, in which the house- 
keeping heat still comes from the work done by some ex- 
ternal driven force but where does the excess heat come 
from?We will show that its origin is just the change of a 
thermodynamic quantity called general internal energy. 



4- Dissipative work and general internal energy 

Notice that the heat conduction only occurs at the spe- 
cific time when the sample trajectory jumps, but which 
manner of energy exchange would happen during the 
time interval as the trajectory stays in the same state? 
It is surely due to the variation of the time-dependent 
transition laws from the external perturbation acting on 
the system. With the classic quantum system in mind, 
we realize that it is just the "dissipative work" called by 
Jarzynski and Crooks [20j, J2, 36, 37,^3^], and rigor- 
ously formulated by Min Qian and the author [33| . 

The dissipative work along the trajectory is defined as 

fHIMIli 



W{X{t),t) = ^kT 



d l og TTj [t) 

dt 



\i=X{t), 



with ensemble average W{t) = — fcT^,-pi(t) '^^°^^'^*'' . 

If the system satisfies the detailed balance conditions 
for all time, i.e. 'Ki{t)qij{t) = ■Kj(t)qji(t), then the tradi- 
tional concept of internal energy exists and both of the 
excess heat and dissipative work contribute to its change, 
which is the First Law undoubtedly [s^. Therefore, we 
believe that the situation will not be essentially different 
even if detailed balance conditions fails. 

Further, we find out that the dissipative work subtract- 
ing the excess heat along the trajectory does not depend 
on the particular "path" taking through the parameter 
space, namely, only depends upon the initial and final 



dU{t) 
dt 



^-Qex{t)+W{t). 



(3) 



It is just the ordinary internal energy for the equi- 
librium canonical ensemble according to the Maxwell- 
Boltzmann's law. Hence here Eq.Q is just the gener- 
alized First Law of thermodynamics. 



5. General free energy and the concept of free heat 

Based on the elementary definition of free energy in 
equilibrium thermodynamics F = U — TS, here we could 
define a general free energy in the same way: 



Fit) = U{t)-TS{t) - fcT Vp.Wlog^, 

I 

which is just the relative entropy of the distribution 
{pi{t)} with respect to another one {7ri(t)} from the 
mathematical point of view fi^ . 

For equilibrium system, it is just the Gibbs free en- 
ergy in a spontaneously occurring chemical reaction at 
constant pressure p and temperature T, and also the 
Helmholtz free energy for systems at constant V and T 
. Its change gives the maximum work, other than pV 
work. Therefore, it is called a "hybrid free energy" by 
Ross, J. dH. 

More important, Schnakenberg has shown that 
it is just the Lyapunov function as well as Prigogine- 
Glansdorff criterion certificating the thermodynamic sta- 
bility for the steady state of the time- independent master 
equation system. This will be revisited in Sec. Ill CI 

Due to the Jensen's equality for the convex function 
— logx, we have F[t) > 0, and the equahty holds if and 
only if Pi{t) = TTiit), for each state i. 

On the other hand, 



dF{t) _ dU{t) dS{t) 



dt 



dt 
W{t) 



T- , 

dt 

(T • epr{t) 



Qhkit)), (4) 



Here we introduce a new concept named Free heat 
Qf{t) = T ■ epr{t) — Qhk{t) identifying the free energy 
change in the form of heat, i.e. 



dFjt) 
dt 



W{t)-Qf{t). 



This concept will play the central role in the extended 
form of the Second Law of Thermodynamics below. 
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B. Energy Balance and External Driven Force 



Former quantitative forms of the Second Law 



The First Law of Thermodynamics is essentiahy an 
extension of the principle of the conservation of energy 
to include systems in which there is flow of heat. But the 
situation becomes more complicated in nonequilibrium 
case, since there may exists some external driven force 
which also pumps energy into the system but does not 
contribute to the change of internal energy 0] ■ 

Denote the work done by the external driven force as 
Edf{t), and we here try to figure out its relationship with 
other quantities defined in the previous sections. 

According to energy balance, one has 



W{t) + Edfit) = 



dU{t) 
dt 



T-hdr{t). 



On the other hand, we have already known that 



Although all the thermodynamic quantities in the pre- 
vious sections could be defined along the sample trajec- 
tory, the Clausius inequality and many other thermody- 
namic constrains related to the Second Law should be 
interpreted statistically through ensemble average. 

Notice that every term in the expression of the entropy 
production rate, i.e. 



epr{t) 



lkJ2iP^ (* ) 9y ) - W (O'Jji ) ) log 



Vi{t)qij{t) 
Pj{t)qji{t) 



is nonnegative, and the equality holds if and only if 
Pi{t)qij(t) = pj{t)qji{t) for each pair of states i and j. 

Then according to Eqs. ^ and we derived several 
inequalities of the differential forms: 



dUjt) 
dt 



-QeAt)+Wit) 



Itot {t) + Qhk{t) + W{t), 



and also Qtot{i) = T ■ hdr{t). 

Therefore, it yields Edf{t) — Qhk{t), which is only 
known to be valid in steady state before. 

Now we understand that there exist totally two kinds 
of external works done on the system, one is the dissipa- 
tivc work W{t) and the other Edf{t) from the external 
driven force. They result in the change of general internal 
energy and the heat dissipation respectively. 



C. Second Law of Thermodynamics 

Traditional Second Law of Thermodynamics has two 
kinds of statements [4^: The Kelvin-Planck statement 
"No process is possible whose sole result is the complete 
conversion of heat into work" and the Clausius statement 
"No process is possible whose sole result is the transfer 
of heat from a colder to a hotter body" . 

Now, a central problem arises: how does these 
thermodynamic laws apply to such a nonequilibrium 
time-dependent process? First, we provide a rigorous 
quantitative approach to the Clausius inequality and 
Helmhotz/Gibbs free energy inequalities based on the 
nonnegativity of entropy production rate for nonequilib- 
rium time-dependent processes. 

Then a new concept "instantaneous reversible process" 
[sot ■^ith zero entropy production rate natually emerges, 
which corresponds to the ideal reversible process involved 
in the classic theory of equilibrium thermodynamics, and 
it will imply that there does not exist any real reversible 
process connecting two different equilibrium states. 

Furthermore, an extended quantitative form of Sec- 
ond Law of Thermodynamics will be developed built on 
the nonnegativity of the new concept "free heat", which 
only appears during time-dependent processes and ac- 
cords well to the phenomenological statement by Oono 
and Panicini 



+ Qtot{t) = T ■ epr{t) > 0, (5a) 

dt 

^^-W{t)~Qhk{t) = -T-epr{t)<0. (5b) 



Eq. (|5ap is just the well-known Clausius inequality 
(dS > — ■%?^), which is rectified to obtain expressions for 
the entropy produced {dS) as the result of heat exchanges 
[Qtot)- And Eq. ((5b|) is a general version of the free 
energy inequality for the amount of work performed on 
the system, since the work values must then be consistent 
with the Kelvin-Planck statement [1^ and forbids the 
systematic conversion of heat to work. 

More precise, the quantity Qhk{t) in Eq. ([5b| van- 
ishes when the detailed balance condition {ni(t)qij{t) = 
TTj(t)qji(t)) holds and then it returns back to the tra- 
ditional Helmhotz or Gibbs free energy inequalities of 
equilibrium thermodynamics depending on whether it is 
a NVT or NPT system jH. In this case, -dF > -W, 
which implies the decrease of free energy gives the max- 
imum dissipative work done upon the external environ- 
ment. 

Their corresponding integral forms are 



TAS + J Qtot{t)dt > 0, (6a) 
AF-y W{t)dt^ J Qhk{t)dt < 0. (6b) 



2. Reversible process 

Reversible process that is beneficial for thermodynamic 
studies, is defined as a process that, once having taken 
place, can be reversed, and in so doing leaves no change 
in either the system or surroundings. However, there still 
leaves a fundamental question: "what is precisely the re- 
versible process that connects two different equilibrium 
states?" Since it is well-known that "only irreversible 
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processes contribute to entropy production", what we 
need here is only to check the condition for which the 
entropy production vanishes. Then we find out that it is 
just equivalent to the concept "instantaneous reversibil- 
ity" [sQ] , which is a natural generalization of equilibrium 
state. 

For the time-dependent process discussed in the 
present article, let epr{t) = for each time t, then we 
find that all the steady distributions {7r(t)} must be in- 
dependent with time t (i.e. irit) = some fixed distri- 
bution tt) and the detailed balance condition holds, i.e. 
■niqij(t) = iTjqji{t). Therefore, during this process, the 
transient state at each time t is just the real equilibrium 
state corresponding to the transition law Q{t). But un- 
fortunately, these equilibrium states are all essentially the 
same as the initial one, only with different time scales, 
i.e. Q{t) — f{t) ■ (5(0) for some function /. In other 
words, there is no real reversible process between two dif- 
ferent equilibrium states, which confirms the well-known 
belief in equilibrium thermodynamics. This reversible 
process is called the "instantaneous reversible process" 
[30|, which could be realized as a generalization of the 
concept of equilibrium state. 

Note that, all the fundamental equations in classic 
equilibrium thermodynamics, such as dU ~ TdS — pdV, 
requires that initial and final equilibrium states be de- 
fined and that there is some reversible path between them 
(4^ . But unfortunately it is not turc, and this is just why 
equilibrium thermodynamics could not be directly gen- 
eralized to the far-from-equilibrium case. 



3. Extended quantitative forms of the Second Law 

Oono and Paniconi Q put forward a phenomenolog- 
ical extended form of the Second Law to noncquilib- 
rium steady states, which claims the irreversibility of the 
process converting work into excess heat when modulo 
house- keeping heat. After that, a quantitative expres- 
sion in Langevin systems was given for the transition 
processes between steady states by Hatano and Sasa [l^ . 

Here we give a rather different but much more general 
derivation, only need to notice that the free heat Qf{t) — 
epr(t) — TQhk{t) > again due to the simple inequality 
log a; < a; — 1 for x > 0: 



Qfit) = T ■ epr{t) - Qhkit) 

p^{t)TTj{t) 
Pj{t)TT,{t) 



Piit)Trj{t) 



j 3^ ' i i j 

= 0, (7) 
Then according to Eqs. ^ and ([3]), we have another 



group of thermodynamic inequalities in the differential 
forms: 



+ Qex{t) - T ■ epr{t) - Qhk{t) > 0, (8a) 

dt 



^^~W{t) = -Qf{t) = -T-eprit)+Qhkit)<0. 

(8b) 

followed by their corresponding integral forms 

TAS + J Qex{t)dt > 0, (9a) 



AF - y" W{t)dt < 0. 



(9b) 



Eq. ((9a|) is the extended form of Clausius inequal- 
ity during any nonequilibrium time-dependent process, 
whose special case is included in Hatano and Sasa's work 
[2^. And Eq. (j9bp is a different general form of free 
energy inequality. It implies the dissipative work value 
must be consistent with the Oono-Paniconi statement of 
the extended Second Law of thermodynamics , which 
forbids the systematic conversion of excess heat to work. 

For equilibrium case, Q^x = Qtot, then they both re- 
turn back to Eq. ([5a|) actually. And then if in steady 
state, then Qf{t) = 0, and this form of the Second Law 
completely disappears. 



III. APPLIED TO CLASSIC 
THERMODYNAMIC PROCESSES 

A. Time-independent case 

1. Equilibrium states and nonequilibrium steady states 

Here we consider the Markov chain with transition 
density matrix Q = {qij} and stationary distribution 
TT = {iTi} satisfying ttQ — 0. 

The first step is to distinguish equilibrium and steady 
nonequilibrium states. As is well known, thermodynamic 
equilibrium is in general maintained through detailed bal- 
ance, which was already known by Boltzmann; and the 
reversibility of a time-independent stochastic process was 
first introduced by Kolmogorov. It has been rigorously 
proved by Min Qian, Minping Qian and their collabora- 
tors that the mathematical essence of these two concepts 
actually turns out to be the same, and the major charac- 
teristic of a nonequilibrium steady state is shown to be 
the positivity of enropy production rate and the appear- 
ance of circulations [15]. 

Note that in the stationary time-independent case, the 
general internal energy, entropy, and free energy are all 
independent of time, and both of the dissipative work 
and excess heat vanish. That may be the reason why 
these concepts have not been found previously, when only 
dealing with steady states. 
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For steady states, we have the important relations 



T ■ hdr — T ■ epr = Qfi^ = 



>0, 



and the equahty holds if and only if at equilibrium states. 

Moreover, the energy balance tells us that the external 
driven force Edf = T ■ hdr, which implies the external 
energy pumped in all changes into heat dissipation [1,|3H|- 
And it is also consistent with the truth that there is no 
external energy source for equilibrium states in which 
hdr = 0. 



2. Relaxation process towards equilibrium states 

Each undriven system will be found to reach a state 
where no further change takes place and it is then said 
to have come to thermodynamic equilibrium. In gen- 
eral, the relaxation to thermodynamic equilibrium will 
involve both thermal and work-like interactions with the 
surroundings, and then which will occur during this re- 
laxion process? 

First, the time- independent property implies the van- 
ishing of dissipative work, i.e. W{t) = 0, but the ex- 
cess heat Qex{i) may not vanish. Second, since the final 
state of the relaxation process is at equilibrium, so the 
external driven force Edf{t) disappears. Consequently, 
Qhk{t) = Edf{t) = 0, and Qtot{t) reduces to Qex{t)- 

However, it will still dissipate heat and have positive 
entropy production rate unless it has already arrived at 
the final equilibrium state. To be more precise, we have 



hdr{t) 



Qtat{t) _ Qex(t) 



therefore. 



dU{t) 
dt 



-QeAt) = -T-hdr{t), 



(10) 



which is just the First Law of Thermodynamics. 
Furthermore, 



dF{t) 
dt 



= W{t)-T-epr{t) + Qnk{t) = -T- 



epr{t) < 0, 
(11) 

hence F(t) is just the Lyapunov function while relaxing 
towards equilibrium states . Until now we know that 
this Lyapunov function is just a directly corollary of the 
Second Law. 



3. Relaxation process towards steady state 

The relaxation process towards steady states has been 
extensively discussed by Glansdorff and Prigogine [1, 0] , 
and then by Schnakenberg for the master-equation sys- 
tems jl4i |. 

Comparing with the previous discussion on the relax- 
ation process towards equilibrium state, we will show 
that the sole difference here only stays within the house- 
keeping heat provided by some specific external driven 
force. 



Here, the First Law of Thermodynamics also becomes 
dU{t) 



dt 



Q ex (i) ; 



and the energy balance reads 



dUjt) 
dt 



T-hdr{t) = Edf{t). 



Combined with Edf{t) — Qhk{t), it is found that the 
whole heat dissipation is from two sources: one is the 
excess heat contributing to the change of general internal 
energy; the other is the housekeeping heat caused by the 
external driven force. 

The extended form of the Second Law now reads 



dFjt) 
dt 



-T ■ epr{t) + Qhk{t) = -Qf{t) < 0, 



thus F{t) serves as a Lyapunov function for this relax- 
ation process [l^, which actually has a solid thermody- 
namic basis now. 



B. Time-dependent case 

Time-dependent processes are causin g m ore and more 
interests from physicists nowadays [23l. l25l. lisl . l46j , and 
it will uncover many important thermodynamic proper- 
ties that originally hidden behind the stationary time- 
independent case. 



1. Cyclic process 

In equilibrium thermodynamics, a thermodynamic cy- 
cle is a series of thermodynamic processes which returns 
a system to its initial state. As a conclusion of cyclic pro- 
cess, all the state variables should have the same value as 
they had at the beginning. Thus AU = AS = AF = 0. 

But variables such as heat and work are not zero over a 
cycle, but rather are process dependent. The First Law of 
Thermodynamics dictates that the net heat input is equal 
to the net work output over any cycle, i.e. / W{t)dt = 

/ Qex{t)dt. 

Hence in this case, the former form of the Second Law 
© gives 



{tot{t)dt = T- I epr{t)dt > 0, (12a) 
W{t)dt > - [ Qhkdt. (12b) 



If one rewrite Eq. (fl2b)) as j'{W{t) + Edf{t))dt = 
J Qtot{t)dt > 0, then it is just the familiar statement 
of traditional Second Law of Thermodynamics "the con- 
version from work to total heat is irreversible" . 

Moreover, the extended form ^ says 
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3. Transitions between steady states 



Qex{t)dt = j Qf{t)dt > 0, 

W{t)dt = / Qex{t)dt > 0, 



(13a) 
(13b) 



confirming the claim that "the conversion from work to 
excess heat is irreversible" Q. In other words, during a 
cyclic process, not only the total heat but also the excess 
heat could only be from the system into the heat bath 
rather than follow the opposite direction. 



2. Transitions between equilibrium states 

Jarzynski provided an expression for the equilibrium 
free energy difference between two configurations of a 
system, in terms of an ensemble of finite-time measure- 
ments of the work performed through switching from one 
configuration to the other (See a recent review [4^ and 
references in). 

In the stochastic-process approach [2l|, [2^ [ll] , we con- 
sider the transition between two equilibrium states real- 
ized in the time interval [0, T] where the detailed balance 
condition is satisfied. In other words, there is no external 
driven force, i.e. Edf{t) = Qhk{t) = 0. 

Then the energy balance becomes 



W{t) 



dU{t) 
dt 



T-hdr{t), 



thus one could realize that it is not very reasonable to call 
the quantity W{t) as "dissipative" , since part of it would 
be stored in the general internal energy rather than really 
convert into heat. 

Notice that when one finishes the task of driving the 
process from time to time T through modulating the 
time-dependent transition density matrix Q{t) (denoted 
as process 1), the system has not reached the final equi- 
librium state yet. Then we should wait until it really 
arrives (denoted as process 2), which is just the "relax- 
ation process towards equilibrium state" with the fixed 
transition density Q{T) described in the previous subsec- 
tions. 

According to Eq. (j6b[) . we have 

AFi < W{t)dt, and AF2 < W(t)dt for process 1 
and 2 respectively. 

Finally, since W{t) = 0, and AF = AFi + AF2 
is just the free energy difference between the initial and 
final equilibrium states, we conclude AF < W{t)dt. 
It is just why we could neglect process 2 when applying 
Jarzynski's work relation in experiments. 

Note that the situation is quite different when regard- 
ing Hitano-Sasa's equality for transitions between steady 
states [2^ . It has not been explicitly pointed out in pre- 
vious works [23, ^a, Wal . See below for details. 



The main contribution of Hatano and Sasa [23, [Sj] is 
to give the first explicit expression for the extended form 
of the Second Law put forward by Oono and Paniconi 
5], during the transition process between steady states. 

They also found out a general definition of entropy 
enlighten from the idea that any proper formulation of 
steady state thermodynamics should reduce to equilib- 
rium thermodynamics in the appropriate limit, so that 
Qex should correspond to the change of a generalized en- 
tropy S. 

However, what they defined is just the general internal 
energy consistent with the First Law of Thermodynam- 
ics rather than the general Gibbs entropy in the present 
article and also in Seifert's recent work [2^. Note that 
the two quantities are always different except for steady 
states, hence the extend form of Second Law of ther- 
modynamics during the course of transition between two 
steady states derived by Hatano and Sasa is not flawed. 

Similar to the preceding transition process between two 
equilibrium states, when one finishes driving the process 
from time to T (also denoted as process 1) through 
varying the time-dependent transition density Q{t) , the 
system has not reached the final steady state yet, and we 
should wait until it arrives (denoted as process 2), which 
is just the "relaxation process towards steady state" 
where the transition density is fixed at Q(T). 

According to Eq. ^ , we get 

/i Qex{t)dt > -ASi, and Qex{t)dt > -AS2 for pro- 
cess 1 and 2 respectively. 

Finally, since AU = AS = A^i -I- AS2 representing 
the energy (entropy) difference between the initial and 
final steady states, we derive that 



Qex{t)dt+ / Qexit)dt > ^AS. 



,{t)dt 



Hatano and Sasa [23| concluded that the equality held 
for an infinitely slow operation in which the system is 
in a steady state at each time during a transition("slow 
process"). 

It is indispensable to emphasize that we could not ne- 
glect process 2 this time when applying Hitano-Sasa's 
identity, because here J2 Qex(t)dt may not be zero, which 
implies that the relaxation process towards the final 
steady state will also contribute to the heat dissipation. 
This critique has already be pointed out by Cohen and 
Mauzerall [47[ , but unfortunately what they criticized is 
the Jarzynski's equality rather than Hatano-Sasa equal- 
ity. In real experiment, this relaxation process may be 
rapid enough and could somehow be omitted. 



C. Summary 

The essential difference between these typical processes 
relies mainly on the signs of the three key thermodynamic 
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quantities: the housekeeping heat, entropy production 
and free heat. And also the dissipative work would dis- 
appear for time-independent processes. See Tab. U for 
details. 



IV. DISCUSSION 

It is the main thesis of this article that we are only at 
the beginning of a new development of theoretical chem- 
istry and physics in which thermodynamic concepts may 
play an even more basic role. "In any case, the num- 
ber of thermodynamic or macroscopic variables is much 
less than the large number of the microscopic degrees of 
freedom. Hence, the transition from a microscopic to a 
macroscopic description involves a drastic reduction of 
the information about the system." [isj 

However, the study of thermodynamics before is 
largely confined to equilibrium states. Although the field 
of "nonequilibrium thermodynamics" has successfully ex- 
tended the 19th century concepts of equilibrium thermo- 
dynamics to the systems that are close to, or near equilib- 
rium, the understanding of far-from-equilibrium systems 
is still poor. 

To investigate these points, stochastic thermodynam- 
ics has stepped much further than other approaches dur- 
ing the last two decades [H, IH, [s^, HH ■ For stochastic 
systems, the central problem is around the extension of 
the Second Law, which originally describes the funda- 
mental limitation on possible transitions between equi- 



librium states. And recently, it has been studied from 
the trajectory point of view, which stimulated the rapid 
emergence of so-called fluctuation theorems [i^, IsH ). 
Most of them are merely valid for specific processes, such 
as steady states or transitions between steady states; 
while the newly developed integral fluctuation theorems 
for total entropy production and housekeeping heat could 
hold for arbitrary time-dependent systems [25, 52]. 

These fluctuation theorems require the definitions of 
thermodynamic functionals along the trajectory, and 
most of the have already done. Thus the main purpose 
of the present article is to investigate the thermodynamic 
laws within them, especially the two quantitative forms 
of the Second Law. 

Based on the stochastic processes, we put forward a 
rather relatively integral theory of nonequilibrium ther- 
modynamics, which may be more definite and convincing 
but more restrictive than the previous phenomenologi- 
cal frameworks. In addition, it would be interesting to 
test experimentally all the quantities and relations, espe- 
cially in nonharmonic time-dependent potentials, where 
one does not expect Gaussian distributions. 
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